Skip to content

[TEST] Golden file test infrastructure for JSON Schema baselines#471

Open
Seth Fitzsimmons (sethfitz) wants to merge 1 commit intodevfrom
golden-json-schemas
Open

[TEST] Golden file test infrastructure for JSON Schema baselines#471
Seth Fitzsimmons (sethfitz) wants to merge 1 commit intodevfrom
golden-json-schemas

Conversation

@sethfitz
Copy link
Copy Markdown
Collaborator

@sethfitz Seth Fitzsimmons (sethfitz) commented Mar 12, 2026

Replaces hand-rolled JSON Schema baseline tests with a shared pytest plugin. Each per-feature test drops from ~35 lines of fixture boilerplate to ~10 lines of declarative call.

What's new

overture.schema.system.testing (in overture-schema-system):

  • assert_golden(actual, golden_path, *, update) -- compare a string against a golden file; on mismatch, raises AssertionError with a unified diff. With update=True, writes instead of comparing.
  • assert_json_schema_golden(model_or_union, golden_path, *, update) -- generate the JSON Schema for a Pydantic model (or discriminated union) and delegate to assert_golden.
  • overture.schema.system.testing.plugin -- pytest plugin that registers the --update-baselines flag and the update_baselines fixture.

Helpers live in system (not a separate test-support package) because system is foundational, the helpers depend on system.json_schema, and themes already depend on system. Mirrors numpy.testing / pandas.testing.

Per-package opt-in

The plugin is activated by each consuming package's pyproject.toml:

[project.entry-points.pytest11]
overture_baselines = "overture.schema.system.testing.plugin"

This keeps --update-baselines from appearing in core, codegen, or cli test runs when those packages are consumed in isolation.

Each baseline test is marked @pytest.mark.baseline so they can be selected or skipped (pytest -m baseline / pytest -m "not baseline"); the marker is registered per-package in [tool.pytest.ini_options].

Test simplification

All 16 per-feature baseline tests reduce to:

@pytest.mark.baseline
def test_building_json_schema(update_baselines: bool) -> None:
    assert_json_schema_golden(Building, GOLDEN, update=update_baselines)

Other changes

  • Makefile: reset-baseline-schemas (which deleted files for regeneration on next run) replaced with update-baselines, which runs pytest --update-baselines -m baseline.
  • pyproject.toml: registers the baseline marker at the workspace level for selective runs.
  • packages/overture-schema-system/README.md: documents the helpers, plugin opt-in, and update workflow.
  • packages/overture-schema-system/tests/testing/test_golden.py: tests for the helpers themselves.

Net: ~435 lines removed, ~416 added (mostly the new helpers, plugin, README section, and their tests); per-test files shrink substantially.

Closes #470

Helpers `assert_golden` and `assert_json_schema_golden` live in
`overture.schema.system.testing`. The pytest plugin at
`overture.schema.system.testing.plugin` registers `--update-baselines`
and the `update_baselines` fixture; consuming packages opt in via
`[project.entry-points.pytest11]`.

Helpers live in `system` rather than a separate test-support package:
`system` is foundational, helpers depend on `system.json_schema`, and
themes already depend on `system`. Mirrors `numpy.testing` /
`pandas.testing`.

Per-package opt-in rather than a single entry point in `system`: keeps
`--update-baselines` from appearing in `core`, `codegen`, or `cli`
test runs when consumed in isolation.

Signed-off-by: Seth Fitzsimmons <seth@mojodna.net>
@sethfitz
Copy link
Copy Markdown
Collaborator Author

Victor Schappert (@vcschapp) I re-did this PR in response to our conversation a few weeks back. Ready to look at again.

@github-actions
Copy link
Copy Markdown

🗺️ Schema reference docs preview is live!

🌍 Preview https://staging.overturemaps.org/schema/pr/471/schema/index.html
🕐 Updated Apr 30, 2026 18:03 UTC
📝 Commit 215756f
🔧 env SCHEMA_PREVIEW true

Note

♻️ This preview updates automatically with each push to this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[TEST] Golden file test infrastructure for JSON Schema baselines

1 participant